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This  report  concerns  the  effects  of  prequestion  and  postquestion  formats  on 
prose  learning  in  computer-assisted  instruction.  Five  experimental  groups 
studied  a set  of  eight  passages  under  different  prequestion-postquestion 
combinations.  Twenty-five  subjects,  volunteers  with  high  school  degrees, 
were  randomly  assigned  to  each  group.  A final  retention  test  had  32  items 
classified  along  two  dimensions:  one  dependent  on  whether  the  inserted 

text  questions  prompted  attention  to  the  item,  and  the  other  dependent  on 
whether  the  item  required  factual  or  inferential  learning.  The  CAI  format 
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facilitated  control  over  exposure  to  questions  and  passages,  as  well  as 
data  collection  on  performance  measures  like  referral  to  passages,  time 
spent  on  inserted  questions,  total  study  time,  and  scores  on  postquestion 
sets.^vA  combination  of  prequestions  and  postquestions  led  to  a 20%  increase 
in  student  study  time  relative  to  the  use  of  either  question  format  alone, 
but  did  not  facilitate  relevant  retention  beyond  the  facilitation  found 
with  just  prequestions  or  just  postquestions.  The  use  of  prequestions 
interfered  with  incidental  learning.  While  equal  to  the  prequestions  format 
in  study  time  and  effect  on  relevant  learning,  the  postquestions  format 
seemed  prefereable  since  it  was  not  accompanied  by  depressed  incidental 
learning.  \ 
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The  Effects  of  Pre-  and  Postquestions  or  Learning 
from  Textual  Material  in  a CAI  Format 

INTRODUCTION 

The  study  described  in  this  report  explores  the  use  of  questions 
to  help  students  learn  from  instructional  materials  presented  on  a 
computer  terminal.  In  particular,  three  ways  of  questioning  students 
about  the  materials  they  are  studying  are  compared.  One  questioning 
format  presents  questions  before  a student  sees  the  text  itself. 

In  this  format  the  student  may  also  review  these  questions  while  he 
studies  the  text.  A second  questioning  format  displays  questions 
only  after  the  student  has  completed  studying  the  text.  In  this 
format  the  student  is  not  allowed  to  review  the  text  when  he  has 
reached  its  questions.  The  final  questioning  scheme  combines  both 
these  formats:  questions  are  displayed  before  the  text  and  whenever 

the  student  wishes  to  review  them;  after  the  student  completes  his 
study  of  the  text  questions  are  presented  again. 

Numerous  investigations  have  been  concerned  with  the  effects  of 
questions  available  during  or  after  study.  The  research  reported 
here  extends  these  efforts  by  considering  the  effects  of  using  questions 
both  during  and  after  the  study  of  textual  materials. 

After  a brief  review  of  relevant  research  on  the  use  of  questions 
as  aids  to  textual  learning,  the  details  and  results  of  our 
investigation  will  be  presented. 

REVIEW  OF  RESEARCH 

Providing  students  with  questions  has  been  shown  to  enhance 
learning  from  textual  materials  (Boyd,  1973;  Rickards  & Di  Vesta, 

1974).  Not  only  is  retention  of  information  relevant  to  the  questions 
facilitated  by  their  use,  but  questions  may  also  facilitate  retention 
of  passage  information  that  does  not  answer  inserted  questions 
(Bruning,  l ‘>f)K ; Rothkopi  & Bisbiseos,  1967).  Rothkopf  (1966, 

1972)  has  attributed  the  effects  of  questions  to  the  control  which 
they  exert  over  the  learner's  inspection  and  processing  of  the  materials. 
The  position  of  questions  relative  to  the  text  is  critical  to  the 
type  of  control  which  is  exerted. 
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Questions  read  before  related  text  segments  direct  the  student's 
attention  to  specific  portions  of  the  text  (Frase,  1970)  , namely 
those  portions  from  which  answers  to  the  questions  can  be  derived. 

As  might  be  expected,  readers  using  such  prequestions  generally 
demonstrate  greater  learning  of  question-relevant  information  than 
do  those  studying  without  access  to  questions.  But  this  positive 
increase  is  frequently  accompanied  by  a relative  decrease  in  the 
learning  of  Information  outside  the  scope  of  the  prequestions. 

Anderson  and  Biddle  (1975)  reviewed  18  studies  on  the  use  of 
prequestions  and  incidental  learning;  they  found  a depressed  level 
of  incidental  learning  reported  in  13  of  the  studies.  Thus  it 
appears  that  prequestions  induce  more  attention  to  selected  text 
segments  than  would  normally  be  given  but  may  also  lead  to  a 
reduction  in  the  attention  given  to  the  remaining  segments. 

The  effects  of  prequestions,  especially  on  incidental  learning, 
may  be  a function  of  the  learner's  understanding  of  the  role  of 
prequestions  relative  to  the  goal  of  instruction  (Duchastel  & 

Brown,  1974).  If  the  learner  believes  the  goal  of  instruction  is 
to  learn  information  pertaining  to  the  prequestions,  his  reading 
is  apt  to  focus  strongly  on  relevant  information  at  the  expense  of 
incidental  information;  if  the  reader  believes  the  goal  of  instruction 
is  to  learn  both  incidental  and  relevant  information,  his  reading  is 
apt  to  be  less  selective.  The  directions  given  students  concerning 
the  purpose  of  prequestions  relative  to  the  purpose  of  instruction 
have  varied  considerably  across  studies.  The  majority  of  investigators 
have  associated  the  two  only  weakly,  if  at  all.  In  view  of  this 
variation  it  is  surprising  that  so  much  evidence  confirms  the  con- 
clusion that  prequestions  depress  incidental  learning.  As  might 
be  expected,  however,  the  effect  has  often  been  small.  In  those 
13  studies  found  by  Anderson  and  Biddle  (1975)  in  which  incidental 
learning  was  depressed,  the  mean  " decrement  in  scores  was  only  2.8. 
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Questions  which  follow  textual  materials  effect  the  learner's 
reading  behavior  differently  than  do  questions  which  precede  materials. 
Since  postquestions  are  seen  after  the  textual  materials  have  been 
completed,  the  materials  are  not  studied  with  the  selective  attention 
found  with  prequestions.  Unlike  prequestions,  postquestions  typically 
facilitate  quest  ion- relevant  retention  without  any  accompanying 
depression  in  question-incidental  retention  (Rothkopf,  1966). 

Indeed  some  researchers  have  found  that  students  given  postquestions 
retain  more  incidental  information  than  do  those  not  given  post- 
questions (Rickards  & l)i  Vesta,  1974;  Rothkopf  6 Bisbiscos,  1967). 

The  effects  of  postquestions  on  question  relevant  information 
can  be  attributed  to  the  reader's  recall  and  review  of  information 
retained  in  memory  that  pertains  to  the  questions  (Anderson  and 
Biddle,  1975).  In  fact,  the  recall  and  review  resulting  from  post- 
questions may  also  involve  information  incidental  but  topically  related 
to  the  question-relevant  information;  McGaw  and  Grotelueschen  (1972) 
and  Rothkopf  and  Billington  (1974)  found  that  postquestions  enhanced 
both  quest  ton- reievant  information  and  informat  ion  only  topically  related 
to  the  relevant  information. 

Interest  in  research  exploring  the  effects  of  postquestions 
on  quest  ion- relevant  information  has  been  overshadowed  by  research 
exploring  the  effects  of  postquest l 'ns  on  incidental  information. 

The  latter  research  has  been  stimulated  by  a number  of  reports 
that  readers  receiving  postquestions  show  superior  incidental 
learning  relative  to  read-only  control  students  (e.g.,  Bruning,  1968; 
Rothkopf,  1966;  Rothkopf  & Bisbiscos,  1967). 

These  indirect  effects  of  post  quest  ions  have  been  shown 
to  be  dependent  on  both  the  type  of  learning  required  to  answer 
the  question  and  the  frequency  with  which  questions  occur  within 
the  text.  For  example,  Felker  and  Dapra  (1975)  explored  the 
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possibility  that  answering  verbatim  rather  thf.n  comprehension  post- 
questions could  differentially  affect  students'  ability  to  apply 
principles  and  concepts  discussed  in  the  text.  The  verbatim  post- 
questions required  recognition  of  exact  text  wording  while  the 
comprehension  questions  required  understanding  of  the  semantic  meaning 
of  text  wording.  Both  question  types  were  matched  in  content  and 
focus.  On  the  problem  solving  test  which  followed  study  of  the  text 
materials,  the  group  using  comprehension  postquestions  outperformed 
the  group  using  verbatim  questions.  In  fact  the  group  with  verbatim 
questions  performed  no  better  than  the  read-only  control  group. 

Rickards  and  Di  Vesta  (1974)  varied  both  the  type  of  learning 
required  to  answer  the  questions  and  the  frequency  with  which 
postquestions  appeared.  These  investigators  used  three  types  of 
postquestions:  one  type  required  literal  verbatim  recall  of  facts, 

another  required  literal  verbatim  recall  of  ideas,  while  the  third 
required  recall  and  organization  of  facts  supporting  ideas.  Students 
encountered  a postquestion  either  after  every  second  paragraph  of 
text,  or  after  every  fourth  paragraph  of  text.  When  questions  appeared 
more  frequently,  both  literal  verbatim  recall  of  ideas  and  recall 
and  organization  of  facts  supporting  ideas  were  found  to  result  in 
significantly  more  learning  of  incidental  ideas  than  literal  verbatim 
recall  of  facts.  Furthermore,  recall  and  organization  of  facts 
supporting  ideas  resulted  in  significantly  more  learning  of  incidental 
facts  than  the  other  two  activities.  But,  when  questions  were  asked 
less  frequently,  incidental  retention  was  the  same  in  all  groups. 

One  explanation  for  the  indirect  effects  of  postquestions 
is  given  in  terms  of  shaping  (Rothkopf,  1972).  According  to 
this  explanation,  a student's  reading  behaviors  preceding  those 
postquestions  which  he  can  answer  are  reinforced  and  maintained, 
while  those  preceding  questions  which  cannot  be  answered  are  not 
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reinforced  and  become  altered.  What  may  result  then  is  that  the 
reader  responding  to  postquestions  will  attend  more  than  he  normally 
would  to  that  type  of  information  to  which  postquestions  are  likely 
to  refer.  Hence,  if  the  incidental  items  on  the  criterion  test 
are  representative  of  the  postquestions,  the  reader  who  received 
postquestions  is  more  likely  to  have  retained  the  information 
they  require. 

With  this  explanation,  a parallel  emerges  between  the  effects 
of  pre-  and  postquestions:  where  prequestions  may  focus  the 

reader's  attention  on  those  text  stimuli  which  provide  answers 
to  the  prequestions,  postquestions  may  focus  his  attention  on 
classes  of  text  stimuli  which  are  likely  targets  for  future 
postquestions.  It  can  be  argued  though  that  the  reader's 
intention  is  related  to  the  former  effect,  while  it  may  not  be 
related  to  the  latter  (Rothkopf,  1972);  whereas  the  student  using 
prequestions  chooses  to  focus  his  attention,  the  student  using 
postquestions  may  not  be  aware  of  his  selective  attention. 

Other  explanations  for  the  indirect  affects  associated  with 
postquestions  have  been  advanced,  including  the  idea  that  post- 
questions provide  practice  for  taking  the  criterion  test. 

Indeed,  the  results  of  Felker  and  Dapra  (1975)  indicate  that  when 
the  demands  of  the  postquestions  parallel  those  of  the  criterion 
test,  incidental  learning  will  be  facilitated. 

In  their  review  of  the  effects  of  questions  on  reading, 

Anderson  and  Biddle  (1975)  conclude  that  postquestions  produce 
greater  learning  than  do  prequestions.  As  might  be  inferred 
from  what  has  been  said  above,  postquestions  have  almost  consistently 
been  shown  to  result  in  better  performance  on  incidental  learning 
than  do  prequestions.  But  also  on  relevant  learning  where  both 
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question  formats  have  been  shown  to  have  positive  results, 
there  is  a trend  for  the  effects  of  postquestions  to  be  stronger 
and  more  consistent  than  those  of  prequestions. 

The  finding  that  postquestions  are  superior  to  prequestions 
in  relevant  retention  is  unexpected  when  one  compares  the  memory 
requirements  involved  in  answering  the  questions.  As  Rickards  (1976) 
has  noted,  in  the  typical  study  investigating  the  effects  of  pre- 
and  postquestions,  the  use  of  prequestions  involves  considerably 
less  reliance  on  memory  than  does  the  use  of  postquest  ions . Stuuents 
are  typically  not  allowed  to  review  the  text  after  they  have  reached 
its  postquestions.  To  correctly  answer  a postquestion  then,  a reader 
is  required  to  remember  one  or  perhaps  several  items  of  information 
from  a text  containing  many  other  items  of  information  which  have 
competed  for  his  attention.  To  answer  a prequestion  the  reader 
has  the  text  at  his  disposal;  furthermore,  his  attention,  can  focus 
on  that  information  relevant  to  the  question.  The  option  of  reviewing 
the  prequestions  while  reading  the  text  has  varied  from  study  to 
study,  but  even  if  the  reader  is  not  given  the  option  of  looking 
back  at  the  prequestions,  the  memory  requirement  imposed  may  be 
considerably  less  than  is  called  for  in  the  postquestion  format. 

Hence,  it  would  seem  that  the  probability  of  learning  question- 
relevant information  is  higher  when  prequestions  are  used  than  when 
postquestions  are  used. 

That  postquestioning  is  in  fact  superior  to  prequestioning 
in  relevant  retention  suggests  that  the  study  behaviors  resulting 
from  postquestions  result  in  more  effective  learning  of  the  question- 
relevant information  than  do  those  resulting  from  prequestions. 

Whereas  the  student  may  more  often  successfully  formulate  an  answer 
to  a prequestion  than  a postquestion,  what  is  learned  by  those 
using  postquestions  is  retained  more  effectively.  Apparently,  the 
advantage  prequestions  have  in  terms  of  memory  requirements  does  not 
compensate  for  the  relatively  less  effective  study  behaviors  which 
they  Induce. 
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A natural  extension  to  the  Investigation  of  pre-  and  post- 

questioning  is  the  exploration  of  the  effects  on  studying  when 

♦ 

the  two  are  combined  in  one  treatment.  With  respect  to  the 
retention  of  relevant  information,  does  the  study  behavior 
invoked  by  the  combination  prove  superior  to  those  behaviors 
associated  with  either  format  alone?  And  what  will  be  the  effects 
of  a combined  format  on  the  retention  of  incidental  information? 

Strongly  supported  answers  to  these  questions  are  not  available 
as  little  research  has  been  conducted  which  explores  the  combined 
questioning  format.  Boyd  (1973)  reports  one  of  the  few  studies  which 
compared  the  combination  with  either  format  alone.  Based  on  Boyd's 
results  and  what  is  known  about  the  effects  of  postquestioning  and 
prequestioning,  some  expectations  about  the  combined  format  will 
be  advanced. 

Boyd  found  that  those  given  both  pre-  and  postquestions  performed 
significantly  better  on  the  retention  of  question-relevant  information 
than  did  those  using  only  prequestions.  The  addition  of  postquestions 
to  prequestions  adds  to  the  reading  behavior  typically  resulting 
from  prequestions  further  processing  of  question-relevant  information. 
It  is  expected  that  this  additional  processing  will  result  in  less 
forgetting  of  that  relevant  information.  Hence,  with  respect  to 
the  retention  of  relevant  information,  the  study  behavior  invoked  by 
the  combination  should  prove  superior  to  that  which  results 
from  prequestions  alone. 

Boyd  also  found  that  those  given  both  pre-  and  postquestions 
performed  significantly  better  on  the  retention  of  question-relevant 
information  than  did  those  using  only  postquestions.  As  has  been 
mentioned  before,  the  reading  behaviors  induced  by  prequestions  may 
result  in  less  effective  learning  of  question-relevant  information 
than  do  those  which  are  induced  by  postquestions.  Indeed,  Boyd's 
data  does  Indicate  that  those  groups  given  only  postquestions  did 
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tend  to  perform  better  on  relevant  retention  than  did  those  groups 
given  only  prequestions.  Apparently  the  addition  of  postquestioning 
to  prequestioning  may  compensate  for  this  relatively  less  effective 
learning  and  result  in  a questioning  format  that  is  superior  to 
postquestioning  alone. 

In  Boyd's  study  incidental  learning  was  depressed  to  the  same 
degree  in  both  the  groups  receiving  only  prequestions  and  those 
receiving  both  pre-  and  postquestions.  As  we  have  mentioned  before, 
prequestions  do  tend  to  focus  the  student's  attention  on  question- 
relevant information.  Apparently  the  addition  of  postquestions  to 
prequestions  does  little  to  alter  this  focusing.  The  effects  of 
a combined  format  on  incidental  learning  will  be  similar  to  those 
of  prequestioning  alone. 

The  primary  purpose  of  our  study  was  to  provide  more  information 
concerning  the  effects  of  inserting  questions  both  before  and 
after  the  text.  The  questions  used  by  Boyd  were  one-word  completions 
requiring  verbatim  recall.  The  questions  used  in  our  study  were 
multiple  choice  typically  requiring  a higher  level  of  reader  compre- 
hension than  is  required  by  verbatim  recall.  Whether  Boyd's  results 
would  be  replicated  with  different  types  of  questions  was  a central 
concern  of  our  investigation. 

A secondary  objective  of  our  study  was  to  pursue  this 
investigation  within  the  context  of  computer-assisted  instruction 
and  thus  extend  the  range  of  materials  to  which  results  might 
apply.  In  computer- assisted  instruction  the  presentation  of  text 
is  a common  instructional  event.  It  was  hoped  that  this  research 
would  provide  authors  of  computer-assisted  instruction  programs  with 
guidelines  concerning  optimum  questioning  formats  to  use  when  textual 
materials  are  to  be  presented. 


METHOD 


Experiment  Participants.  One  hundred  and  twenty-five  undergraduates 
from  a local  four  year  college  participated  in  the  study.  The  students 
were  paid  volunteers  solicited  through  classroom  announcements  and 
advertisements  in  the  campus  paper. 

Each  student  studied  a sequence  of  eight  unrelated  passages, 
using  one  of  five  different  question  formats.  After  studying  the 
passages,  students  were  requested  to  take  a 32  item  multiple  choice 
retention  test,  with  four  questions  relating  to  each  passage. 

Materials.  The  eight  passages  were  selected  from  those  used  in  the 
reading  comprehension  subtest  of  the  Scholastic  Aptitude  Test  (SAT). 

The  passages  chosen  had  been  published  in  brochures  describing  the 
test  and  were  no  longer  being  used  on  forms  of  the  SAT.  They  each 
averaged  about  400  words  in  length.  The  topics  covered  by  the 
passages  were  diverse,  with  four  passages  discussing  topics  in  the 
physical  and  biological  sciences,  and  four  discussing  topics  in  the 
social  sciences  and  humanities.  This  range  in  topics  was  intended 
to  facilitate  implications  for  prose  learning  in  general  rather  than 
for  a specific  subject  matter. 

Questions  from  the  SAT  associated  with  the  passages  provided 
about  60%  of  the  questions  required  in  the  study.  The  remaining 
questions  were  constructed  by  the  investigators  in  consultation  with 
professional  examiners  whose  full-time  responsibility  was  test 
development.  All  questions  were  multiple  choice  in  format. 

The  questions  required  five  types  of  textual  processing: 

(1)  comprehension  of  a supporting  idea,  explicitly 

stated  in  the  passage, 

(2)  comprehension  of  a major  idea  only  indirectly 

expressed  in  the  passage. 
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(3)  completion  of  an  inference  made  in  the  passage, 

(4)  application  of  a generalization  found  in  the  passage 

to  a particular  instance, 

and  (5)  evaluation  of  the  logic  of  the  author's  discussion. 

Questions  of  the  first  type  were  classified  as  lower  order  items. 
Questions  of  the  remaining  four  types  were  labeled  higher  order  items. 
The  majority  of  the  higher  order  items  were  of  types  2 and  3.  A 
total  of  64  questions  were  used,  four  lower  and  four  higher  order 
items  for  each  passage. 

A better  understanding  of  the  question  types  can  be  gained  through 
some  examples.  The  following  question  based  on  a passage  about  the 
life-style  of  birds  assesses  comprehension  of  a supporting  idea 
explicitly  stated  in  the  passage  (type  1): 

It  is  essential  that  birds  have  an  efficient 
respiratory  system  because  they 

(A)  are  generally  small  in  size 

(B)  fly  in  rarefied  atmsophere 

(C)  have  great  muscular  development 

(D)  must  be  mentally  alert  to  insure 
survival 

*(E)  lead  an  active  existence 

The  passage  segment  from  which  the  question  was  derived  reads: 

Respiration  is  efficient  and  indeed  must  be  to 
sustain  the  high  metabolic  activity. 

This  particular  item  paraphrases  the  idea  as  explicitly  stated.  Both 
questions  using  paraphrased  restatements  and  verbatim  statements  were 
actually  used  as  lower-order  items. 

An  example  of  an  item  assessing  comprehension  of  a main  idea 
(type  2)  is  taken  from  a passage  which  critically  reviews  Plato's 
Repub  lie . Although  the  actual  purpose  of  the  work  is  not  stated, 
the  author  provides  enough  information  about  its  contents  and  the 
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manner  In  which  it  is  written  to  allow  the  following  question  to  be 
answered : 


It  is  inferred  that  Plato's  Republic  is  a work  written 
principally  for  the  purpose  of 

(A)  providing  citizens  with  a guide  to  the  best 
possible  life 

(B)  changing  existing  methods  of  education 

*(C)  convincing  readers  that  the  rule  of  the  few  is 
preferable  to  the  rule  of  the  many 

(D)  convincing  the  populace  of  biological  differences 
among  classes 

(E)  encouraging  people  to  overthrow  existing  governments 

In  a passage  concerning  the  development  of  anesthetics,  the 
following  statement  is  made: 

"Curare  and  its  derivatives  are  being  replaced  by  a 
number  of  synthetics  that  are  more  specific  in 
action  and  predictable  in  effect." 

Completion  of  an  inference  made  in  the  statement  is  required  by  the 
following  type  3 question: 

It  can  be  inferred  that  a disadvantage  of  using  curare 
derivatives  as  anesthetic  agents  is  that  they 

(A)  are  dangerously  explosive 

(E)  are  difficult  to  manufacture 

*(C)  can  produce  unexpected  effects 

(P)  are  not  as  powerful  as  newer  drugs 

(E)  cannot  be  combined  with  other  drugs 

A passage  concerning  the  evolution  and  operation  of  cooperatives 
makes  the  statement: 

A consumers'  cooperative  sells  its  goods  at  the  prevailing 
competitive  prices  in  order  to  avoid  conflict  with  other 
retailers. 
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Application  of  the  rule  suggested  in  this  statement  concerning 

how  a cooperative's  prices  are  determined  is  required  in  the  following 

type  4 question: 

Which  of  the  following  actions  would  probably  be 
taken  by  a cooperative  in  response  to  a competitive 
price  cut  by  neighboring  stores? 

(A)  It  would  maintain  its  prices  and  its  dividend 
at  their  regular  levels. 

(B)  It  would  maintain  its  prices  at  their  regular 
level  and  increase  its  dividend. 

*(C)  It  would  cut  its  prices  to  equal  those  of  Its 
competitors . 

(D)  It  would  cut  its  prices  to  undersell  its 
competitors . 

(E)  It  would  dissolve. 

An  example  of  the  last  item  type  comes  from  an  historical  passage 

concerning  Puerto  Rican  politics.  In  the  passage  the  author  uses  the 
term  "unstable  marriage."  The  meaning  of  the  term  can  be  determined 
by  comprehending  implications  of  statements  which  surround  it.  The 
following  question  assesses  whether  the  reader  has  understood  these 
implications : 

The  author  probably  describes  the  Alianza  as  an  "unstable 
marriage"  in  order  to  indicate  that 

(A)  it  had  not  been  officially  sanctioned  by  the 
government 

(B)  one  faction  was  deliberately  undermining  the 
united  party 

(C)  one  group  had  been  misinformed  about  the 
party's  platform 

(D)  there  had  not  been  unanimous  consent  to  the 
coalition 

*(E)  it  included  two  irreconcilable  factions 

Each  passage  had  four  higher-order  and  four  lower-order  ques- 
tions associated  with  it.  The  higher-  and  lower-order  questions  for 
a passage  were  randomly  paired,  and  the  pairs  randomly  assigned  a 
number  from  1 to  4.  The  item  pairs  were  then  assigned  as  inserted 
prequestions,  inserted  postquestions,  and/or  retention  test  items  on 
the  basis  of  the  numbers  thev  received.  Table  1 summarizes  the1 


the  basis  of  the  numbers  they  received.  Table  1 summarizes  the 
assignment  scheme  of  a passage  set  of  item  pairs  relative  to  the 
live  format  groups. 
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Tahir  I 

Quest  ion  Ass  i gnmcnt  Relative 
to  Format  Croups 


Form at 

Ass  ignment  s 

Preques t i ons 

Postquest  ions 

Retention  Items 

CONTROL 

none 

none 

1,2 

PRE 

1,3 

none 

1,2 

POST 

none 

i , i 

1 ,2 

PRF/POSTlR) 

1 ,3 

1 ,3 

1,2 

PRE/POST  (P) 

1 ,3 

l , 

1 ,2 

Note.  Numbers  refer  to  item  pair  assignments. 

The  following  remarks  can  all  be  inferred  from  Table  1;  they 
concern  the  questions  seen  by  the  various  groups  on  a given  passage. 
Question  pairs  l and  2 were  used  to  measure  retention  for  all  five 
groups.  For  all  groups  except  CONTROL,  question  pair  1 measured 
relevant  learning,  and  question  pair  2 measured  incidental  learning. 
Question  pairs  1 and  3 were  seen  by  all  groups  except  CONTROL:  as 

prequestions  by  group  PRE,  as  postquest  ions  by  group  POST,  and  as 
both  pre-  and  postquestions  by  group  PRK/POST(R).  The  group 
PRE/POST(P)  saw  question  pairs  1 and  3 as  prequestions,  but  only 
pair  3 was  repeated  in  the  postquestioning;  they  saw  the  final 
question  pair,  4,  for  their  remaining  two  postquestions.  Thus, 
none  of  this  group's  postquestions  were  repeated  on  the  retention  test. 

Those  stems  of  questions  used  as  both  pre-  and  postquestions  were 
revised  so  that  the  item  could  be  stated  as  an  open-ended  question 
requiring  the  student  to  formulate  his  own  response.  This  was  the 
form  in  which  prequestions  were  presented,  while  postquestions  were 


presented  in  multiple  choice  format.  For  example,  of  the  questions 
given  as  illustrations  of  the  types  of  questions  used  In  the  study, 

4 were  selected  for  use  as  both  prequestions  and  postquestions.  As 
postquestions  they  appeared  in  the  multiple  choice  format  presented 
above;  as  prequestions  they  appeared  as  follows: 

According  to  the  passage,  why  is  it  essentia]  that  birds 
have  an  efficient  respiratory  system? 

What  disadvantage  of  using  curare  derivatives  as 
anesthetic  agents  may  be  inferred  from  the  passage? 

It  may  be  inferred  from  the  passage  that  a cooperative's 
response  to  a competitive  price  cut  by  neighboring 
stores  would  be  ...? 

What  is  the  author  probably  implying  with  the  use  of 
"unstable  marriage"  to  describe  the  Alianza? 

The  passages  were  placed  into  t lie  topical  groupings  of  physical/ 
biological  sciences  and  social  sciences/humanities.  Item  analysis  data 
from  previous  test  administrations  was  available  for  those  passage 
items  borrowed  from  the  SAT.  The  data  was  used  to  order  the  passages 
in  each  topical  set  in  terms  of  the  difficulty  of  their  associated 
questions,  the  passage  with  easiest  questions  coming  first.  This 
method  of  passage  ordering  was  chosen  because  it  was  felt  that  presenting 
progressively  more  challenging  questions  was  typical  of  instructional 
materials.  Two  passage  presentation  orders  were  used  in  the  study; 
both  orders  maintained  the  topical  grouping  orders.  One  presentation 
order  placed  all  the  social  science/humanities  passages  first,  followed 
by  the  physical/biological  science  passages,  while  the  second  alternated 
between  the  two  topical  groups,  with  the  easiest  social  science/humanities 
passage  first. 

All  passages  and  inserted  questions  were  presented  to  the  students 
via  a PLATO  computer  terminal.  The  retention  test  was  in  paper  and 
pencil  format.  The  computer  presented  the  study  materials  as  follows: 
all  prequest  tons  for  a passage  were  contained  on  one  display,  separate 
from  the  passage;  the  passages  were  contained  on  one  or  two  displays, 
depending  on  their  length;  each  postquestion  appeared  on  a separate 
display. 

Through  the  use  of  selected  kevs  on  the  terminal  keyboard, 
students  controlled  the  rate  at  which  materials  were  presented. 
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Those  having  access  to  prequestions  were  presented  the  display  of 
a passage's  prequestions  prior  to  the  presentation  of  the  passage. 

While  they  studied  the  passage,  those  using  prequestions  were 
free  to  review  them  as  often  as  desired.  Students  were  not  required 
to  overtly  respond  to  prequestions.  Those  having  access  to  postquestions 
were  not  allowed  to  review  the  passage  after  they  had  requested 
the  display  of  its  postquest  ions . Those  using  postquestions  were 
required  to  answer  the  postquestions  by  pressing  the  terminal  key 
designating  their  alternative  choice  (a,b,c,d,  or  e) . Students  were 
not  allowed  to  review  a postquestion  once  they  had  seen  it  and  had 
gone  on  to  another  display. 

Question  Formats.  Five  question  formats  were  explored.  They  will  be 
denoted  CONTROL,  PRF , POST,  PRK/POST(R) , and  PRE/POST(P) . 

CONTROL:  Those  using  the  CONTROL  format  saw  no  inserted 

questions  during  their  study  of  the  passages. 

PRF,:  Students  studying  the  passages  using  the  PRE 

format  received  prequestions  before  each  passage. 

Four  prequestions  preceded  each  passage,  two  higher 
order  and  two  lower  order.  The  prequestions  were 
open-ended  requiring  the  student  to  formulate  his  own 
response. 

POST:  Students  studying  the  passage  using  the  POST  format 

received  postquestions  after  each  passage.  Four  questions 
followed  each  passage.  These  were  the  same  questions  that 
those  using  the  PRE  format  saw  before  each  passage,  but 
were  stated  as  multiple  choice  rather  than  open-ended 
quest  ions . 

PRF./POST(R):  Students  studying  the  passage  using  this 

format  received  both  the  prequestions  received  by  the 
PRE  students  and  the  postquestions  received  by  the  POST 
students.  The  ' R'  in  the  above  name  denotes  the  fact 
that  the  postquestions  repeated  the  prequestions,  i.e., 
there  was  a one-to-one  correspondence  between  the  open- 
ended  prequestions  and  the  multiple  choice  postquestions. 

Hall  of  those  questions  seen  by  the  student  groups  described  above 
also  appeared  on  the  retention  test,  2 from  each  passage,  1 higher 
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order  and  1 lower  order.  Those  questions  repeated  on  the  retention 
test  appeared  in  the  multiple  choice  form. 

The  PRE  group  dealt  only  with  the  open-ended  forms  of  the 
inserted  questions  repeated  on  the  retention  test.  Because  of  this, 
if  we  foune  that  the  PRE/POST(R)  group  outperformed  the  PRE  group, 
some  of  that  improvement  could  be  attributed  to  the  former  group's 
practice  with  criter ion-iike  items.  To  separate  out  the  effects  of 
criterion  test  practice  provided  by  the  postquestions  from  the  effects 
of  further  relevant  informatipn  processing  resulting  from  the  post- 
questions in  the  PRE/POST(R)  format,  the  following  format  was 
included  in  the  study. 

PRE/POST(P) : For  each  passage,  students  studying  the 

materials  using  this  format  received  the  prequestion  set 
used  by  the  PRE  and  PRE/POST(R)  groups,  but  received 
only  two  of  the  postquestions  seen  by  the  POST  and  PRE/ 
POST(R)  groups.  The  two  postquestions  presented  were 
those  lower  order  and  higher  order  items  not  repeated 
on  the  retention  test.  In  addition,  two  more  questions 
were  added  to  the  each  postquestion  set,  1 higher  order 
and  lower  order,  that  were  not  used  by  any  of  the  other 
groups  and  not  included  on  the  retention  test.  (This 
accounts  then  for  the  16  items  whose  function  was  not 
identified  above.) 

Thus  those  in  the  PRE/FOST(P)  group  did  not  receive  as  post- 
questions any  of  the  items  which  later  appeared  on  the  retention 
test.  The  'P'  in  the  above  name  denotes  the  fact  that  the  post- 
questions  provided  only  practice. 

The  PRE/POST(P)  group  allowed  us  to  explore,  not  only  the  effects 
of  practice,  but  also  the  effects  on  reading  behavior  of  postquestions 
referring  to  prequestion-incidental  information.  It  was  expected 
that  prequestions  alone  would  cause  students  to  focus  their  attention 
on  the  prequestioned  information  to  such  an  extent  that  the  learning 
of  incidental  information  would  be  impaired.  If  postquest  ions 
repeating  the  prequestions  were  added  (i.e.,  the  PRE/POST(R)  format). 
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the  focusing  would  not  be  altered.  But,  if  the  postquestion  set 
included  new  questions,  would  the  student  not  only  search  for  pre- 
question answers,  but  also  prepare  for  the  new  questions  by  studying 
more  carefully  other  parts  of  the  text?  The  investigators  did  expect 
that  those  in  the  PRE/POST(P)  group  would  indeed  attend  more  to 
prequestion- incidental  Information  that  did  those  in  the  PRE  or 
PRE/POST(R)  groups. 

Procedures . A PLATO  terminal  was  installed  on  the  campus  of  the 
local  college.  Student  participants  were  signed-up  for  individual 
appointments  of  approximately  90  minutes.  They  were  randomly  assigned 
to  a question  format.  Twenty-five  students  were  included  in  each 
group . 

When  a student  arrived  for  his  appointment,  he  was  seated  in 
front  of  the  terminal  and  asked  to  read  some  directions.  The  directions 
discussed  the  purpose  of  the  experiment  and  the  use  of  the  terminal. 

They  also  described  the  question  format  that  the  student  would  be 
using  while  he  studied  the  passages.  The  directions  read  by  the 
PRE  and  PRE/P0ST(P)  groups  are  included  in  the  appendix  of  this 
report.  An  attempt  was  made  to  maintain  parallel,  if  not  identical, 
directions  for  each  group. 

The  directions  for  those  students  not  using  the  CONTROL  format 
indicated  that,  after  their  study  of  the  materials,  they  would  be 
given  a test,  and  on  that  test  would  be  the  questions  they  had  seen 
as  inserted  questions.  There  was  no  indication  given  of  new  questions 
appearing  on  the  final  test.  In  order  to  confirm  that  students  did 
not  expect  new  questions  to  appear  on  the  retention  test,  a number 
of  participants  were  asked,  after  they  had  completed  the  retention 
test,  if  they  had  expected  new  questions  to  appear  on  the  test. 

All  confirmed  that  they  had  not  expected  new  questions. 

The  directions  for  CONTROL  students  specified  that  they  should 
learn  as  much  as  they  could  from  the  passages.  They  were  warned  that 
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a test  would  follow  their  study,  and  on  that  test  would  be  questions 
asking  them  to  recall  facts  and  ideas  stated  in  the  passages  and  make 
inferences  based  on  things  that  had  been  said  in  the  passages. 

After  a student  completed  the  directions  the  principal  investigator 
showed  him  how  to  use  the  keys  he  would  need  to  work  through  the 
materials.  Also,  under  the  guidance  of  the  investigator,  the  student 
practiced  the  key  presses  on  a sample  passage  which  included  those 
inserted  questions  appropriate  to  the  format  to  which  the  student 
was  assigned. 

All  students  were  given  as  much  time  as  they  needed  to  complete 
the  materials.  But,  once  they  had  left  a passage  to  go  on  to  the  next 
passage,  they  were  not  allowed  to  return  to  it.  To  provide  some 
motivation  for  studying  the  materials,  students  were  told  that  if 
they  answered  more  than  40%  of  the  retention  test  questions  correctly, 
they  would  receive  an  extra  monetary  compensation  for  their  participation 
in  the  study.  In  reality  all  students  received  the  same  reimbursement. 

RESULTS  AND  DISCUSSION 

Table  1 presents  group  means  and  standard  deviations  for  the 
retention  test  subscores  derived  from  the  following  sets  of  items: 
the  eight  higher  order  items  that  also  appeared  as  inserted  questions 
(higher  order  relevant  learning-HRL) , the  eight  lower  order  items  that 
also  appeared  as  inserted  questions  (lower  order  relevant  learning-LRL) , 
the  eight  higher  order  items  that  did  not  appear  as  inserted  questions 
(higher  order  incidental  learning-HIL) , and  the  eight  lower  order 
items  that  did  not  appear  as  inserted  questions  (lower  order  in- 
cidental learning-LIL) . Also  Included  in  the  table  are  the  means 
and  standard  deviations  for  total  relevant  retention  (HRL  + LRL) , 
total  incidental  retention  (H1L  + LIL) , and  total  overall  retention 
(HRL  + LRL  + H1L  + LIL).  The  data  suggest  that,  with  respect  to  relevant 
learning,  the  POST  group  outperformed  those  groups  given  prequestions. 
Furthermore,  scores  on  the  higher  order  questions  account  for  most  of 
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the  relevant  learning  difference  between  the  POST  group  and  those 
groups  who  received  prequestions.  An  unanticipated  outcome  in  the 
data  was  the  poor  performance  of  the  PRE/POST  with  practice  group 
in  incidental  learning.  The  expectation  that  this  group  would  out- 
perform both  the  PRE  group  and  the  PRE/POST  with  repetition  group 
in  incidental  learning  was  not  supported.  A more  systematic  analysis 
of  the  retention  test  scores  will  begin  with  some  orthogonal  contrasts. 


Table  1 

Results  on  Retention  Test:  Subscores  and  Total  Score 


RELEVANT 

LEARNING 

INCIDENTAL  LEARNING 

OVERALL 

TREATMENT 

HRL(8) * 

LRL(8) 

T0TAL(16) 

HIL(8) 

LIL(8) 

T0TAL(16) 

TOTAL(32) 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

POST 

6.04 

1.27 

6.32 

.94 

12.36 

4.36 

1.66 

5.32 

1.34 

9.68 

2.34 

22.04 

3.66 

PRE 

5.36 

1.85 

6.20 

1.53 

11.56 

3.00 

3.56 

1.83 

5.12 

1.59 

8.68 

2.85 

20.24 

5.52 

PRE/POST(R) 

5.28 

2.17 

6.04 

1.67 

11.32 

3.35 

3.72 

1.62 

4.44 

1.83 

8.16 

2.97 

19.48 

5.63 

PRE/POST (P) 

5.20 

1.98 

5.76 

1.62 

10.96 

3.03 

3.28 

1.84 

4.68 

1.77 

7.96 

3.14 

18.92 

5.65 

CONTROL 

4.96 

2.01 

1.54 

10.08 

3.07 

4.24 

1.83 

5.12 

1.51 

9.36 

2.77 

19.44 

5.12 

Number  In  parentheses  represents  the  maximum  score  possible 


Orthogonal  Contrasts  for  Relevant  and  Incidental  Learning 

Through  orthogonal  contrasts  an  attempt  was  made  to  answer  the 
following  questions  concerning  the  retention  of  question-relevant  informa- 
tion: 

(1) 

(2) 

(3) 


How  does  the  performance  of  students  with  access  to  questions 
compare  with  that  of  students  without  access  to  questions? 

How  does  the  performance  of  students  given  only  post- 
questions compare  with  that  of  students  given 
prequest  ions? 

How  does  the  performance  of  students  given  postquestions 
as  well  as  prequestions,  either  in  the  form  of  practice 
or  repetition,  compare  with  the  performance  of  those 
given  only  prequest  Ions? 


i 
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(4)  When  both  pre-  and  postquestions  are  used,  how  do 
students  given  postquestions  providing  repetition 
differ  in  performance  from  those  given  postquestions 
providing  practice? 


The  contrasts  underlying  these  questions  reflect  some  of  our  initial 
expectations  about  relative  group  performance  in  relevant  retention. 

The  first  contrast  allows  us  to  determine  whether  access  to  questions 
facilitated  the  retention  of  question-relevant  information.  Based 
on  the  findings  of  other  investigators,  we  expected  that  questioning 
would  prove  helpful.  As  was  discussed  in  the  review  of  research, 
postquestioning  has  typically  been  superior  to  prequestioning  in 
facilitating  relevant  retention;  by  comparing  the  retention  of  those 
who  received  only  postquestioning  with  those  who  received  prequestioning 
with  or  without  postquestioning,  the  second  contrast  allows  us  to 
access  whether  the  superiority  of  postquestioning  is  supported  by 
our  data.  The  third  contrast  permits  the  comparison  of  the  combined 
formats  with  prequestioning  alone.  Since  we  expected  the  combined 

formats  to  add  to  those  behaviors  typically  resulting  from  pre- 
questioning further  processing  of  question-relevant  information  and/or 
criterion  test  practice,  we  did  expect  those  using  the  combined 
format  to  outperform  those  using  only  prequestions.  The  last  contrast 
allows  us  to  compare  the  effects  of  criterion  test  practice  with  the 
effects  of  further  relevant  information  processing. 

Table  2 presents  the  t-values  for  these  contrasts. 
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Table  2 

Orthogonal  Contrasts  Relevant  Learning 


Contrast 

Number 

Groups  Compared 

t Value 

Two  tailed 
t-Probability 
(df  = 120) 

i 

POST,  PRE,  PRE/F0ST(R),  PRE/P0ST(P) 

VS.  CONTROL 

2.27 

.04 

2 

PRE,  PRE/POST (R) , PRE/POST (P) 

VS.  POST 

-1.61 

.12 

3 

PRE/POST (R),  PRE/FOST (P) 

vs.  PRE 

- .59 

4 

PRE/ POST (P) 

VS  - PRE/POST (R) 

- .44 

MS  Error 

Within  Groups  - 8.42 


As  can  be  seen  from  Table  2,  the  results  of  contrast  1 indicate 
that  those  given  questions  outperformed  those  without  access  to 
questions:  providing  students  with  questions  did  improve  question- 

relevant retention.  Although  the  retention  test  subscores  suggested 
that  those  using  only  postquestions  learned  more  than  those  using 
formats  which  included  prequestions,  the  results  of  contrast  2 indicate 
that  this  difference  is  not  statistically  significant  at  the  standard 
.05  level.  As  indicated  by  the  results  of  the  third  contrast,  the  reten- 
tion of  those  receiving  postquestions  along  with  prequestions  was  Just 
about  the  same  as  those  receiving  only  prequestions.  Furthermore,  as 
can  be  seen  in  the  fourth  comparison,  there  was  no  significant  difference 
between  the  pre/post  combinations.  Since  the  third  contrast  suggests 
that  the  addition  of  postquestions  to  prequestions  has  little  effect 
on  relevant  retention,  it  would  seem  to  follow  that  it  makes  little 
difference  on  relevant  retention  whether  the  postquestions  provide 
both  criterion  test  practice  and  repetition  or  just  criterion  test 
practice . 

With  respect  to  incidental  learning,  our  expectations  concerning 
relative  group  performance  are  reflected  in  the  following  questions: 

(1)  How  does  the  performance  of  students  given  question 
formats  Including  prequestions  compare  with  that  of 
those  given  formats  not  including  prequestions? 
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(2)  How  does  the  performance  of  students  given  only  post- 
questions compare  with  that  of  those  given  no 
questions? 

(3)  Of  those  students  given  prequestions,  how  does  the 
performance  of  those  who  receive  postquestions  referring 
to  information  incidental  to  the  prequestions  compare 

to  that  of  those  who  do  not? 

(4)  How  do  students  given  prequestions  differ  in  performance 
from  those  given  both  prequestions  and  postquestions, 
where  the  postquestions  refer  again  to  the  prequestioned 
information? 

The  contrast  Inferred  in  the  first  question  allows  us 
to  assess  whether  prequestioning  resulted  in  depressed  incidental  learning 
in  comparison  to  those  formats  which  did  not  include  prequestioning. 

Based  on  the  findings  of  other  investigators,  we  expected  prequestioning 
to  focus  study  behavior  to  such  an  extent  that  incidental  learning  would 
be  lowered.  The  comparison  suggested  in  the  second  question  between 
the  group  given  only  postquestions  and  the  read-only  control  group 
attempts  to  uncover  whether  postquestioning  had  the  'indirect'  effect 
of  facilitating  incidental  retention.  The  third  contrast  allows  us  to 
assess  whether,  in  a combined  format,  the  use  of  postquestions  which 
do  not  match  prequestions  facilitates  incidental  retention  relative  to 
the  use  of  only  prequest  ions  or  both  prequestions  and  matching  post- 
questions. This  question,  of  course,  lias  previously  been  answered 
in  our  observations  about  the  retention  subscores  of  Table  1;  the 
PRE/POST(P)  group  did  not  do  as  welt  as  did  either  the  PRE  or  PRE/POST(R) 
groups  oti  incidental  retention.  The  last  contrast  addresses  the 
expectation  that  the  effect  on  incidental  learning  of  inserting  questions 
both  before  and  after  the  text  will  be  similar  to  the  effect  of 
prequestions  alone. 

Table  3 presents  the  t-values  for  these  contrasts. 
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Contrast 

Number 


Table  3 

Orthogonal  Contrasts:  Incidental  Learning 

Two-tailed 

Croups  Compared  t-Value  t-Probability 

(df  - 120) 


1 PRE,  PRE/POST (R) , PRE/P0ST(P)  VS.  POST,  CONTROL  -2.43 

2 POST  VO.  CONTROL  .40 

3 PRE,  PRE /POST (R)  VS.  PRE/POST(P)  - -66 

4 PRE  vs.  PRE/POST (R)  -65 

MS  error 

Ulthln  Groups  - 7.99 


.02 


The  results  of  the  first  contrast  confirm  the  expectation  that 
prequestions  depress  incidental  learning.  The  results  of  the  second 


contrast  indicate  that  students  receiving  only  postquestions  did 
about  as  well  as  the  control  students  on  incidental  learning;  there  is 


no  evidence  that  postquestions  had  the  indirect  effect  of  facilitating 


incidental  retention  above  that  associated  with  simply  reading  a 
passage.  Since  the  PRE/P0ST(F)  format  did  not  result  in  facilitating 
incidental  learning  relative  to  the  PRE  or  PRE/P0ST(R)  format,  there 
is  no  evidence  that  the  PRE/P0ST(P1  format  induces  a more  generalized 


reading  strategy  than  is  produced  by  prequestions  alone  or  prequestions 
with  repeated  postquestions.  The  results  of  the  last  contrast  support 
the  contention  that  the  processing  of  incidental  Information  induced 
by  prequestions  is  little  effected  by  the  addition  of  matching 
postquestions . 


Before  proceeding  to  further  analyses  of  the  data,  a summary  of 
the  major  results  disclosed  thus  far  will  be  given.  With  respect  to 
relevant  retention,  it  was  found  that  those  given  questions  performed 
significantly  better  than  did  those  not  given  questions.  There  was 
no  evidence,  though,  that  having  access  to  questions  both  before  and 
after  text  was  superior  to  either  using  only  prequestions  or  post- 
questions. In  fact,  trends  in  the  relevant  learning  subscores 
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indicated  that  those  given  only  postquestions  outperformed  those 
given  formats  including  prequestions.  Although  this  difference  was 
not  statistically  significant  it  is  in  agreement  with  the  finding  of 
other  investigators  that  the  effect  of  postquestioning  on  relevant 
learning  is  stronger  than  that  of  prequestioning.  With  respect  to 
incidental  retention,  it  was  found  thSit  prequestioning  did  depress 
incidental  learning.  On  the  other  hand,  there  was  no  evidence  that 
the  POST  format  facilitated  incidental  learning  relative  to  the 
CONTROL  format. 

Prequestions  vs.  Postquestions 

More  information  about  the  question  formats  can  be  gained  by 
considering  just  the  groups  CONTROL,  POST,  PRE  and  PRE/POST(R) . 

Doing  so  allows  one  to  conceive  of  the  study  as  involving  two  factors, 
where  presence  of  prequestions  and  absence  of  prequestions  constitute 
the  levels  of  one  factor  and  presence  of  postquestions  and  absence 
of  postquestions  constitute  the  levels  of  the  other  factor.  A two 
factor  analysis  of  variance  using  these  factors  was  performed  for  the 
dependent  variables  incidental  learning  and  relevant  learning.  In 
this  analysis  the  main  effect  of  prequestions  involves  comparing  the 
average,  effect  of  the  PRE/POST(R)  and  PRE  treatments  with  that  of 
the  POST  and  CONTROL  treatments,  while  the  main  effect  of  postquestions 
involves  comparing  the  average  effect  of  the  PRE/POST(R)  and  POST 
treatments  with  that  of  the  PRE  and  CONTROL  treatments. 

The  summary  table  for  the  analysis  of  variance  using  the 
dependent  variable  of  Incidental  retention  appears  in  Table  4. 

As  can  be  seen,  the  only  significant  difference  is  the  main  effect 
of  prequestions,  reconfirming  that  formats  including  prequestions 
depress  incidental  learning  relative  to  the  no  prequestioning  formats. 


r 
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Table  A 

Analysis  of  Variance  Summary  Table: 

Presence  of  Prequestions  x Presence  of  Postquestions 
Dependent  Variable:  Incidental  Retention 


Source  of 
Variation 

Sum  of 
Squares 

df 

Mean 

Squares 

F 

Significance 

PREQUESTIONS 

30.25 

1 

30.25 

A. 02 

.05 

POSTQUESTIONS 

.25 

1 

.25 

.03 

INTERACTION 

A.  41 

1 

A .41 

.59 

WITHIN  CELLS 

721.99 

96 

7.52 

The  summary  table  for  the  two— factor  analysis  of  variance 
conducted  for  the  dependent  variable  of  relevant  retention  appears 
as  Table  5. 


Table  5 

Analysis  of  Variance  Summary  Table 
Presence  of  Prequestions  x Presence  of  Postquestions 
Dependent  Variable:  Relevant  Retention 


Source  of 
Variation 

Sum  of 
Squares 

df 

Mean 

Squares 

F 

Significance 

PREQUESTIONS 

1.21 

1 

1.21 

.15 

— 

POSTQUESTIONS 

26.01 

1 

26.01 

3.16 

.08 

INTERACTION 

39.69 

1 

39.69 

A. 83 

.03 

WITHIN  CELLS 

789.19 

96 

8.22 

L J 
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As  can  be  seen  from  Table  5,  a significant  Interaction  was 
revealed  by  the  analysis.  The  appearance  of  an  Interaction  indicates 
that  the  two  treatment  means  of  one  factor  behave  differently  under 
different  levels  of  the  other  factor.  Based  on  the  results  reported 
previously,  this  interaction  is  expected;  we  have  already  observed 
that  the  addition  of  postquestions  to  a format  which  has  no  prequestioning 
results  in  facilitating  relevant  retention,  while  the  addition  of 
postquestions  to  a format  which  has  prequestioning  has  little  effect 
on  relevant  retention.  Figure  1 graphically  illustrates  how  the 
effects  of  the  presence  or  absence  of  postquestions  depend  on  the 
presence  or  absence  of  prequestions. 

Figure  1 


Interaction  of  Prequestions  and  I’ostquest  ions 
in  Relevant  Learning 


Relevant 

Learning 


Additional  insight  Into  the  Interaction  is  gained  through  tests  of 
simple  main  effects.  These  tests  are  summarized  In  Table  6. 


Table  6 


Tests  of  Simple  Main  Effects 
for  Relevant  Learning 


Source 

SS 

df 

MS 

F 

Signif icance 

PREQUESTIONS  (PRE) 

1.21 

1 

1.21 

PRE  vs . CONTROL 

27.38 

1 

27.38 

3.33 

.10 

POST  vs.  PRE/POST(R) 13.52 

1 

13.52 

1.65 

.25 

POSTQUESTIONS  (POST) 

26.01 

1 

26.01 

POST  vs.  CONTROL 

64.98 

1 

64.98 

7.90 

.01 

PRE/POST (R)  vs.  PRE 

.72 

1 

.72 

.09 

— 

INTERACTION 

39.69 

1 

39.69 

W.  CELL 

789.19 

96 

8.22 

As  shown  in  Table  6,  the  difference  between  the  POST  and  CONTROL 
groups  is  significant,  while  that  between  the  PRF  and  CONTROL  groups 
is  not  significant  at  commonly  accepted  levels.  There  was  no  significant 
difference  in  relevant  retention  between  the  PRF,  and  PRE/POST(R) 
groups.  Finally,  the  prequestion/postquestion  combination  was  not 
significantly  different  from  postquestions  alone. 

How  is  the  lack  of  significance  in  the  difference  between  the 
PRE  and  CONTROL  means  consistent  with  the  finding  presented  earlier 
that  questioning  has  a significant  positive  effect  on  relevant 
retention?  You  will  recall  that  the  conclusion  presented  earlier  was 
based  on  a more  powerful  t-test  which  compared  the  effect  of  the 
CONTROL  group  to  the  average  effect  of  all  questioned  groups, 
including  the  relatively  higher  scoring  POST  treatment.  Hence  it  is 
certainly  possible  that  the  two  tests  might  result  in  differing 
conclusions.  While  the  ANOVA  does  not  establish  the  superiority 
of  the  prequestioned  groups  relative  to  the  CONTROL  group,  it  does 
show  that  the  POST  format  was  reliably  better  than  no  questioning; 
this  supports  the  contention  that  postquestioning  has  a stronger 
impact  on  relevant  learning  than  does  prequestioning. 


Repeated  Measures  ANOVA: Question  Format,  Type  of  Learning,  Level  of  Question 

Still  more  information  about  the  question  formats  was  obtained 
through  a 5 x 2 x 2 analysis  of  variance.  The  factors  were  (a)  the 
five  question  formats,  (b)  the  two  types  of  learning  (incidental  vs. 
relevant),  and  (c)  the  two  levels  of  questions  (higher  order 
v.s.  lower  order),  with  repeated  measures  on  the  last  two  factors. 

Table  7 contains  a summary  of  this  analysis. 


Table  7 

Analysis  of  Variance  Summary  Table 
Question  Format  x Type  of  Learning  x Level  of  Question 


Source  of 
Variation 

Sum  of 
Squares 

df 

Question  Format 

37.29 

4 

Sub j . W.  Groups 

802.44 

120 

Type  of  Learning 

193.44 

1 

Type  of  Learning  x 
Question  Format 

25.19 

4 

Type  of  Learning  x 
Subj . W.  Groups 

182.12 

120 

Level  of  Question 

82.42 

1 

Level  of  Question  x 
Question  Format 

7.65 

4 

Level  of  Question  x 
Subj.  W.  Groups 

191.68 

120 

Type  of  Learning  x 
Level  of  Question 

10.66 

1 

Type  of  Learning  x 
Level  of  Question 
Question  Format 

X 

3.13 

4 

Type  of  Learning  x 
Level  of  Question 
Subj.  W.  Groups 

X 

196.96 

120 

Mean 

Squares  £ Significance 

9.32  1.39  

6.69 

193.44  127.46  .001 

6.30  4.15  .005 

1.52 

82.42  51.60  .001 

1.91  1.20  

1.60 

10.66  6.49  .025 

.78  .48  

1.64 
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As  the  table  indicates,  there  was  no  reliable  difference  between 
the  treatments  in  total  retention.  Relevant  learning  was  significantly 
greater  than  incidental  learning;  performance  on  the  lower  order 
questions  was  significantly  higher  than  performance  on  the  higher  order 
questions.  The  other  two  significant  findings  were  the  interactions: 

(1)  type  of  learning  x question  format  and  (2)  type  of  learning  x 
level  of  question.  We  will  further  explore  both  of  these  interactions. 

An  understanding  of  the  first  interaction  is  aided  by  the  following 
plot  of  the  relevant  retention  and  incidental  retention  subscores  for 
the  various  question  formats. 

Figure  2 

Question  Format  and  Retention: 

, „ Relevant  and  Incidental  Performance 

13.00 

12.00 

RETENTION  11.00 
SUBSCORE 

10.00 

9.00 

8.00 

7.00 

POST  PRE  PRE/  PRF./  CONTROL 
POST (R)  P0ST(P) 

QUESTION  FORMAT 
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As  shown  in  Figure  2,  there  is  a notable  difference  in  relevant 
and  incidental  learning  throughout  the  first  four  formats,  but  not  in 
the  fifth,  the  CONTROL  format.  This  is  consistent  with  our  expecta- 
tions since  the  distinction  between  relevant  and  incidental  learning 
does  not  really  apply  in  the  control  group.  In  fact,  tests  of  simple 
main  effects  indicated  that  relevant  learning  was  significantly 
higher  than  incidental  learning  at  the  .001  level  for  all  groups 
except  the  CONTROL.  There  was  no  significant  difference  between 
relevant  and  incidental  learning  in  the  control  group  (p  ^ .25). 

Tests  of  simple  main  effects  were  also  performed  to  assess 
whether  there  were  differences  among  the  incidental  retention  means 
or  differences  among  the  relevant  retention  means.  Neither  test  was 
significant  at  the  .05  level,  although  the  test  comparing  the  relevant 
retention  means  approached  significance  (p^.08). 

We  turn  now  to  the  significant  type  of  learning  x level  of  question 
interaction.  Tests  of  simple  main  effects  revealed  that  relevant 
retention  was  greater  than  incidental  retention  for  both  higher  and 
lower  learning  (p^.001);  tests  of  simple  main  effects  also  indicated 
that  lower  order  learning  was  greater  than  higher  order  learning  for 
both  incidental  and  relevant  retention  (p?^.005).  The  interaction 
detected  was  due  to  the  fact  that  the  increment  in  performance  for 
the  relevant  higher  order  items  relative  to  the  incidental  higher  order 
items  was  greater  than  that  associated  with  the  lower  order  items 
(Figure  3) . One  possible  interpretation  of  this  is  that  higher  order 
learning  is  aided  more  by  the  use  of  questions  than  is  lower  order 
learning.  But  competing  with  this  interpretation  are  two  less 
interesting  explanations. 


il 


Figure  ' 

Type  of  Learning  and  Retention; 
Higher-order  and  Lower-order  Performance 


RETENTION 

SUBSCORE 


If  the  difference  in  difficulty  between  the  items  assigned  to  measure 
relevant  higher  order  learning  and  those  assigned  to  measure  incidental 
higher  order  learning  was  greater  than  the  difference  in  difficulty 
between  the  relevant  and  incidental  lower  order  item  sets,  an  interaction 
of  the  type  represented  in  Figure  3 might  be  expected.  As  item  assign- 
ment was  random,  there  is  no  a priori  reason  for  believing  that  this 
was  in  fact  the  case.  But  a look  at  the  control  group's  subscores 
(Table  1)  will  provide  more  concrete  information  about  the  differences 
in  difficulty  between  the  item  subsets. 

In  the  control  group  the  relevant  items  were  in  no  way  distinguished 
from  the  incidental  items;  hence,  for  this  group,  the  overall  difficulty 
of  items  measuring  relevant  learning  should  match  the  overall  difficulty 
of  the  items  measuring  incidental  learning.  Table  1 indicates  that 
the  subscore  for  those  lower  order  items  measuring  relevant  learning 
was  equal  to  the  subscore  for  those  lower  order  items  measuring 
incidental  learning.  But  the  corresponding  higher  order  subscores 
were  not  equal.  A t-test  was  performed  to  test  the  hypothesis 
that  the  higher  order  item  subsets  were  in  fact  equal  in  difficulty. 

The  hypothesis  could  not  be  rejected  at  the  .05  level.  Hence  differen- 
tial item  difficulty  does  not  appear  to  be  a tenable  explanation  for  the 
interaction . 
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Another  possible  reason  for  the  observed  interaction  is  the  tendency 
of  subjects  to  perform  near  ceiling  on  the  relevant  items.  Both 
higher  and  lower  order  relevant  means  are  artificallv  depressed  by 
the  ceiling  effect.  Furthermore,  since  lower  order  items  are  easier 
than  higher  order  items,  the  lower  order  mean  is  more  sensitive  to 
the  effect  than  is  the  higher  order  mean.  This  difference  in 
sensitivity  may  account  for  all,  or  at  least  part,  of  the  difference 
in  increments  between  the  levels  of  questions  across  types  of  learning. 

The  major  results  brought  to  light  by  the  above  analyses  of 
variance  will  now  be  summarized.  Of  the  relevant  learning  means,  only 
those  of  the  POST  and  CONTROL  groups  were  reliably  different.  The 
significant  difference  between  the  POST  and  CONTROL  means  was  established 
through  a direct  comparison  of  the  two  treatments  while  exploring  for 
simple  main  effects  (Table  b) . A test  comparing  all  5 relevant  learning 
treatment  means  was  pertormcd  as  an  offshoot  of  the  Question  Format  x 
Type  of  Learning  x Level  of  Question  ANOVA.  This  test  indicated  that 
there  were  no  significant  differences  between  any  of  the  treatment 
means.  Since  the  test  comparing  all  5 means  was  less  powerful  than 
the  test  comparing  the  POST  and  CONTROL  groups  directly,  it  is  not 
lnconsistint  that  the  former  test  does  not  reestablish  the  results  of 
the  latter. 

In  comparison  to  students  not  having  access  to  the  prequestions, 
those  who  did  experienced  depressed  incidental  learning.  This  finding 
was  supported  earlier  in  an  orthogonal  contrast  (Table  3)  and  again  in 
the  significant  main  effect  of  the  Prequestions  x Postquestions 
ANOVA  presented  in  Table  4.  But  the  comparison  of  incidental  learning 
means  done  in  conjunction  with  the  Question  Format  x Type  of  Learning 
x Level  of  Question  ANOVA  detected  no  reliable  differences  between 
the  means.  Again  the  results  of  the  three  tests  can  be  considered 
consistent  since  the  orthogonal  contrast  and  the  two  factor  ANOVA 
provided  more  powerful  comparisons  of  those  receiving  prequestions 
and  those  not  receiving  prequestions. 
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The  Question  Format  x Type  of  Learning  x Level  of  Question 
ANOVA  detected  no  reliable  differences  in  the  overall  learning  means 
of  the  3 treatments.  Furthermore,  this  analysis  indicated  that 
lower  order  learning  was  greater  than  higher  order  learning  for  all 
groups,  and  relevant  learning  was  greater  than  incidental  learning 
for  all  groups  except  the  CONTROL.  A Type  of  Learning  x Level  of 
Question  interaction  was  supported  by  the  analysis.  One  possible 
explanation  of  the  interaction  is  that  higher  order  learning  benefited 
more  by  the  insertion  of  questions  than  did  lower  order  learning. 

Data  collected  on  the  time  students  spent  studying  the  materials 
is  summarized  in  Table  8.' 


Table  8 

Mean  Time  Spent  Inspect ing  Different  Parts  of  the  Materials 


Pro- 
quest i oils 

Post- 

questions 

Passages 

Total 

F orma t 

a 

n 

M 

sp 

M 

M 

SD 

1 

M 

SD 

CONTROL  11 

— 

— 

— 

— 

. A S 

10.72 

3d  .43 

10.7  2 

PR  1 11 

1 1 ..(.*! 

4.04 

— 

— 

•VO  . 7 5 

1 1 . Id 

34  . 1 7 

14.31 

POST  .’1 

— 

— 

1"  . la 

* . 7 

. 4(> 

8 . 3 

37.80 

1 1.20 

I’K  K/ POST  ( R ) .’1 

u..’ 

1.88 

1M  . 0 S 

7.  IS 

1 7 . 9 1 

88 

69  . 1 1 

16 .04 

PKP,/  POST(P  1 .’1 

0. 

1 . (>  1 

18.67 

a .hi' 

la  . -0 

10.40 

62.4b 

14.97 

Note.  Mean  seoret.  are  in  minutes. 

■'Number  of  subjects  within  each  treatment  tor  which  complete 
time  records  were  available.  All  means  within  a treatment  are 
based  on  the  subjects  with  complete  records. 


1 

When  a computer  failure  occurred  while  a student  was  studying  the 
materials,  the  time  history  collected  to  that  point  was  lost.  The 
proctor  reentered  the  student  into  the  program  at  that  point  in  the 
materials  where  ho  had  been  prior  to  the  failure.  Hence  complete 
records  were  not  available  for  all  students. 
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For  each  set  of  means,  a one-way  analysis  of  variance  was  used  to 
assess  their  differences.  When  a significant  difference  was  detected, 
mean  comparisons  were  performed  using  the  Tukey-B  test  (Winer,  1972). 

There  were  no  significant  differences  in  the  postquestion  means 
of  the  POST,  PRF./POST(R) , and  PRE/POST(P)  groups,  or  the  passage 
means  of  the  five  groups.  There  was  a significant  difference  in  the 
time  the  PRE,  PRE/POST(R)  and  FRE/POST(P)  groups  spent  on  the 
prequestions  (p^.01).  At  the  .05  level,  the  PRE  mean  was  significantly 
greater  than  the  PRE/P0ST(F)  mean,  while  the  PRE/POST (R)  mean  was  not 
reliably  different  from  either  of  the  other  two  means.  There  was 
also  a significant  difference  in  the  total  time  the  five  groups  took 
to  complete  the  materials  (p^'.OOl).  At  the  .05  level,  the  CONTROL 
mean  was  significantly  smaller  than  the  other  means,  while  the  PRE/P0ST(R) 
mean  was  significantly  greater  than  the  POST  and  PRE  means.  No  reliable 
differences  could  be  detected  between  the  PRE/F0ST(R)  and  PRE/P0ST(P) 
total  time  means,  or  between  the  PRE,  POST,  and  PRE/P0ST(P)  total  time 
means . 

The  results  concerning  the  time  groups  spent  studying  the  different 
parts  of  the  materials  allow  us  to  reject  one  possible  explanation  for 
why  we  did  not  replicate  the  finding  of  Boyd  (1973)  that  the  combination 
of  pre-  and  postquestioning  results  in  superior  relevant  retention 
relative  to  either  format  alone.  As  the  time  data  indicates,  those 
in  the  PRE/P0ST(R)  group  spent  about  the  same  amount  of  time  on  the 
prequestions  and  passages  as  did  those  in  the  PRE  group;  they  spent 
about  the  same  amount  of  time  on  the  postquestions  and  passages  as 
did  those  in  the  POST  group.  Furthermore , the  PRE/P0ST(R)  group 
spent  significantly  more  time  on  the  task  than  did  either  the  PRE 
group  or  the  POST  group.  Based  on  study  time  alone,  one  might  expect 
the  relevant  learning  of  the  PRE/POST(R)  group  to  have  been  stronger 
than  that  of  the  PRE  or  POST  groups.  But  it  was  not,  and  it  is  to 
speculations  about  why  it  was  not  tltat  we  now  turn. 


■*--  
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Discussion  of  Relevant  Retention  and  Inserted  Questions 

Since  the  PRE  group  did  as  well  as  the  PRE/POST(R)  group  on 
relevant  retention,  we  may  assume  that  the  additional  processing 
provided  by  postquestioning  had  little  effect  on  relevant  learning. 

Why  Boyd  was  able  to  see  an  effect  while  we  were  not  may  be  due  to  the 
difference  between  the  studies  in  how  students  were  allowed  to  use 
prequestions.  In  our  study,  those  using  prequestions  could  review 
them  while  they  studied  the  text.  In  Boyd's  study  students  were  not 
allowed  to  review  prequestions  once  they  began  the  text.  Because  of 
their  free  access  to  prequestions,  it  is  likely  that  those  in  our 
prequestioned  groups  were  better  able  to  formulate  answers  to  the 
prequestions  while  they  studied  the  text  than  were  those  in  Boyd's 
study.  It  may  be  that  the  impact  of  postquest  ions  in  a combined  format 
depends  on  how  well  students  have  been  able  to  formulate  answers  to 
prequestions  prior  to  seeing  the  postquestions . Free  access  to 
prequestions  may  result  in  learning  the  question  relevant  material 
so  well  that  the  additional  processing  resulting  from  answering  the 
questions  again  adds  little  to  learning. 

At  the  onset  of  our  study,  we  did  expect  the  PRE/POST(R)  format 
to  result  in  superior  relevant  retention  relative  to  the  PRE  format. 

But  we  were  less  certain  about  how  the  PRE/POST(R)  format  would  compare 
to  the  POST  format.  The  textual  processing  associated  with  post- 
questioning typically  results  in  more  relevant  learning  than  does  that 
associated  with  prequestioning.  Since  the  processing  induced  by  the 
PRE/POST(R)  format  is  just  an  extension  of  that  induced  by  prequestions, 
it  is  difficult  to  speculate  which  format  will  result  in  superior 
relevant  retention.  The  addition  of  postquestions  to  prequestions  in 
Boyd's  study  appears  to  have  compensated  for  the  less  effe  five 
learning  resulting  from  prequestioning:  those  using  the  combined 

format  outperformed  those  using  only  postquestions.  Our  results,  of 
course,  indicate  that  the  two  formats  were  not  significantly  different 
in  the  relevant  learning  they  promoted.  Why  we  did  not  replicate 
Boyd's  finding  may  be  due  to  differences  in  the  stimulus  materials 
used  in  the  two  investigations. 
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As  was  discussed  in  the  review  of  research,  Rickards  (1976) 
has  observed  that  the  formulation  of  answers  to  inserted  questions 
may  require  less  reliance  on  memory  when  the  inserted  questions 
appear  as  prequestions  rather  than  as  postquestions.  If  the  stimulus 
materials  create  a situation  in  which  this  advantage  results  in  the 
prequestion  group  answering  more  of  the  inserted  questions  correctly, 
those  using  prequestions  may  learn  more  relevant  information  than  do 
those  using  only  postquestions.  Assuming  those  using  prequestions  do 
answer  more  inserted  questions  correctly  than  do  those  using  postquestions, 
if  review  of  the  prequestions  is  restricted,  those  using  a combined 
format  will  be  even  more  likely  to  outperform  those  using  only  postquestions. 

Boyd's  materials  did  result  in  those  using  combined  formats 
more  successfully  answering  the  inserted  questions.  Evidence  for  this 
is  provided  by  comparing  scores  on  inserted  postquestions  for  those 
using  the  combined  formats  with  scores  on  inserted  postquestions  for 
those  using  only  postquestions.  In  Boyd's  study,  those  using  both 
pre-  and  postquestions  scored  significantly  higher  on  the  inserted 
postquestions  than  did  those  using  only  postquestions. 

In  our  investigation,  how  the  PRE/1’0ST(R)  and  POST  groups  compared 
on  the  performance  of  inserted  questions  is  summarized  in  Table  9. 

Listed  are  the  means  for  the  following  item  subsets: 

(1)  higher  order  postquestions  not  repeated  on  the  retention 

test , 

(2)  lower  order  postquestions  not  repeated  on  the  retention 

test, 

(3)  higher  order  postquestions  repeated  on  the  retention 

test,  and 

(4)  lower  order  postquestions  repeated  on  the  retention  test. 


t-test  indicated  that  there  were  no  significant  differences  between 
the  groups  on  any  of  the  four  subscore  means.  Apparently,  the  advantage 
prequestions  may  have  relative  to  postquestions  in  placing  less  demand 
on  students'  memory  did  not  play  a significant  role  in  the  stimulus 
materials  of  this  investigation."^ 
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Table  9 


POST  and  PRF./POST(R)  Inserted  Question  Means  as 
a Function  of  Appearance  of  Inserted  Questions  on  the 
Retention  Test,  and  Retention  Means  of  Repeated  Items 


Item  Subset 

PKK/POST(K) 

Ret  ont i on 

Inserted 
Quest  ions 

Re  tent  ion 

M 

M 

SD 

M 

SI) 

M 

SD 

Higher-Order  Items 

Repea  t ed 

S.hS 

1.23 

6.17 

1.23 

5.65 

1 .87 

5.35 

2.12 

Not  Repeated 

A.  52 

1.47 

— 

— 

4.48 

1.76 

— 

— 

Lower-Order  Items 

Repeated 

6.17 

1 . 19 

6.3d 

.94 

5.96 

1 .69 

6.00 

1.70 

Not  Repeated 

-S.91 

1.41 

— 

— 

6.48 

1.31 

— 

— 

Note.  All  data  is  based  on  the  23  subjects  in  each  group  for 
which  inserted  question  data  were  available. 

Speculations  concerning  the  retention  of  relevant  information 
experienced  by  students  receiving  only  prequestions,  only  postquestions, 
or  the  combination  of  prequestions  and  postquestions  will  now  be 
summarized.  It  seems  that  if  students  are  allowed  to  review  prequestions 
while  they  are  studying  the  materials,  the  addition  of  postquestions 
to  the  treatment  will  not  improve  relevant  retention.  If  this  review 
is  not  allowed,  the  further  processing  of  question-relevant  information 
provided  by  postquestions  may  improve  relevant  retention.  The  addition 
of  prequestions  to  a postquestion  format  will  typically  not  improve 
relevant  retention.  Materials  whicli  place  great  demands  on  student 
memory  may  provide  an  exception  to  this  generalization.  Such  demands 
may  be  made,  for  example,  by  passages  which  are  laden  with  factual 
information  and  Inserted  questions  requiring  retrieval  of  selected 
facts.  Rickards  (1976)  provides  a concrete  example  of  such  stimulus 
materials.  If  the  materials  do  make  demands  on  students'  memory, 


"The  question  arises  as  to  whether  the  relative  advantage  of  prequestions 
is  a function  of  whether  postquestions  require  recall,  as  in  Boyd's  study, 
or  recognition,  as  in  our  investigation.  Since  we  did  not  compare  the  two 
postquestion  types,  we  have  no  data  upon  which  to  base  an  answer  to  this 

question. 
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the  combined  format  may  be  superior  to  only  postquestioning  but  may 
not  be  superior  to  only  prequestioning.  As  we  have  observed  before, 
if  students  are  allowed  to  review  prequestions  while  they  study  the 
text,  additional  postquestioning  may  add  little  to  relevant  retention. 

Textual  Processing 

At  several  points  in  our  discussion  we  have  used  the  conjecture 
that  the  relevant  information  processing  resulting  from  postquestioning 
may  differ  from  that  induced  by  prequestions.  Some  evidence  supporting 
this  conjecture  is  provided  by  comparing  the  following  scores  within 
the  POST  and  PRE/POSl (R)  groups:  scores  on  those  inserted  postquestions 

which  were  also  retention  test  items  and  scores  on  those  same  questions 
when  they  appeared  as  retention  items  (Table  9). 

For  the  23  students  in  the  PRE/POST(R)  group  with  complete  records 
on  responses  to  postquestions,  the  mean  score  on  the  retention  test 
for  higher  order  items  used  as  postquestions  was  5.35.  For  these 
students  the  mean  score  on  the  retention  test  for  those  lower  order 
items  used  as  postquestions  was  6.00.  Two-tailed  t-tests  indicated 
that  neither  subscore  was  significantly  different  from  its  inserted 
question  counterpart,  although  the  difference  in  the  higher  order 
subscores  approached  significance  (p^.09). 

For  the  23  students  in  the  POST  group  with  complete  records  on 
responses  to  postquestions,  the  mean  score  on  the  retention  test  for 
those  higher  order  items  used  as  inserted  questions  was  6.17.  For 
these  students,  the  mean  score  on  the  retention  test  for  those  lower 
order  items  used  as  inserted  questions  was  6.39.  While  the  difference 
in  the  lower  order  Item  subscores  was  not  significant,  the  higher 
order  subscores  were  significantly  different  at  the  .002  level. 

While  no  significant  changes  seemed  to  occur  in  the  PRF./P0ST(R) 
group  between  the  two  question  answering  periods,  the  POST  group  seemed 
to  make  gains  in  relevant  higher  order  learning.  If  the  two  groups 
had  processed  the  question  relevant  information  in  the  same  way,  one 
would  expect  the  changes  between  the  two  question  answering  periods 
to  be  similar.  Since  they  are  not,  some  support  is  provided  for 
the  alternative  hypothesis. 


CONCLUSIONS 


Some  conclusions  regarding  the  question  formats  used  in  the 
study  will  now  be  drawn.  The  combined  format  of  prequestioning  and 
postquestioning,  with  the  postquestions  repeating  the  prequestions, 
is,  in  general,  an  inefficient  way  to  promote  relevant  learning. 

In  our  study  it  resulted  in  an  approximately  20%  increase  in  student 
study  time  relative  to  the  use  of  only  pre-  or  postquestioning  but 
did  not  result  in  improved  relevant  retention  relative  to  these  formats. 

The  combined  format  of  prequestioning  and  postquestioning, 
with  the  postquestions  containing  new  items,  is  ineffectual.  The 
appearance  of  prequestion-incidental  postquestions  does  not  seem  to 
induce  more  regard  for  incidental  information  than  results  in 
groups  using  only  prequestions  or  prequestions  and  matching  postquestions. 
Furthermore,  as  indicated  by  the  group's  spending  significantly  less 
time  with  the  prequestions  than  did  the  other  prequestioned  groups, 
the  new  postquestions  seem  to  induce  less  regard  for  the  learning  of 
relevant  information. 

Time  spent  studying  the  materials  with  the  use  of  only  post- 
questions was  not  significantly  different  from  time  spent  with  the 
use  of  only  prequestions.  Also  there  were  no  significant  differences 
in  the  relevant  learning  promoted  by  the  two  formats.  Yet  postquestioning 
may  be  preferable  to  prequestioning  for  two  reasons: 

(1)  Postquestioning  typically  has  a stronger  positive  Impact  on 
higher  order  relevant  learning  than  does  prequestioning 

and  (2)  Prequestioning  depresses  incidental  learning  relative 

to  postquestioning. 

As  the  analysis  of  time  spent  studying  the  textual  materials 
indicated,  those  using  only  prequest  ions , only  postquestions,  or  no 
questions  at  all  did  not  differ  reliably  in  the  tim-.  they  took  to 
study  the  passages.  But  those  who  studied  with  no  questions  did  take 
considerably  less  study  time  than  the  others  when  time  spent  on 
inserted  questions  was  added  to  passage  time.  The  learning  advantages 
resulting  from  inserted  questions  must  be  weighed  against  the  extra 
time  involved  in  their  use. 
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The  increments  we  observed  in  relevant  learning  which  resulted 

3 

from  inserted  questions  are  not  large.  Relative  to  the  control 
group,  relevant  retention  increased  about  12%  in  those  groups  using 
only  prequestions  or  postquestions.  Yet  the  use  of  prequestions  or 
postquestions  increased  the  time  taken  to  complete  the  materials  by 
approximately  50%.  Was  this  a worthwhile  investment  of  students'  time? 

With  different  textual  materials  and  different  types  of  questions 
the  above  figures  would  undoubtably  change,  but  the  above  question 
would  still  be  relevant.  The  instructor  who  contemplates  the  use  of 
questions  should  assess  how  difficult  it  is  for  the  learner  to  recognize 
what  it  is  he  is  suppose  to  retain  from  the  materials  and  how  critical 
it  would  be  to  Instructional  continuity  if  certain  things  from  the 
passage  were  not  retained.  Such  considerations  will  help  the  instructor 
decide  whether  a,  perhaps  slight,  increase  in  the  probability  of 
relevant  learning  is  worth  the  increased  study  time. 


3 

The  increments  observed  are  typical  of  those  commonly  found  by 
Investigators.  In  those  studies  reviewed  by  Anderson  and  Biddle  (1975) 
which  compared  postquestioning  to  no  questioning,  the  mean  % increment 
in  relevant  learning  for  the  postquestioned  groups  was  13.2.  The 
percent  increment  in  the  present  study  for  POST  was  14.3.  In  those 
studies  reviewed  which  compared  prequestioning  to  no  prequestioning, 
the  mean  X increment  in  relevant  learning  for  the  prequestioned  groups 
was  10.8.  The  percent  increment  in  the  present  study  for  PRE  was  9.3. 
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Student  Directions 


The  study  you  will  be  a part  of  concerns  methods  to  help  students  in 
learning  from  written  materials  presented  on  a computer  display.  The 
particular  method  we  are  interested  in  is  the  use  of  questions  related 
to  the  materials. 

You  will  be  asked  to  read  eight  passages  presented  on  a computer 
terminal.  Before  each  passage  you  will  be  given  four  questions  requesting 
information  which  is  either  contained  in  or  can  be  inferred  from  the  passage. 
(From  now  on  these  questions  will  be  called  prequestions.)  The  prequestions 
are  open-ended.  They  are  designed  to  direct  your  learning  as  you  read  the 
passage.  If  after  studying  the  passage  you  feel  reasonably  confident  about 
your  answers  to  prequestions,  then  you  have  a sign  that  you  have  studied 
the  passage  adequately. 

When  you  have  completed  reading  all  eight  passages  and  working  through 
their  prequestions,  you  will  be  asked  to  take  a final  test.  The  test  will 
contain  the  prequestions  you  have  seen,  but  now  they  will  be  restated  in 
multiple-choice  format.  The  test  will  measure  how  much  you  remember  from 
the  information  you  were  directed  to  learn  from  the  prequestions.  It  will 
also  conclude  the  study. 

You  will  not  be  allowed  to  take  any  notes  while  you  are  reading  through 
the  materials. 

The  passages  are  about  400  words  each.  Each  concerns  different  and 
unrelated  topical  areas.  Topics  chosen  are  from  the  humanities,  social 
sciences,  physical  sciences,  and  biological  sciences. 

(As  you  read  through  the  rest  of  the  directions,  you  may  wish  to 
refer  to  the  Key  Board  Directions  summarized  on  the  next  page.) 

Passage  prequestions  will  be  displayed  on  the  terminal  before  you  see 
their  related  passage.  After  reading  and  familiarizing  yourself  with  these 
prequestions,  you  can  go  on  to  the  passage  by  pressing  the  'NEXT'  key  on 
the  computer  terminal  keyboard.  Thereafter,  anytime  you  wish  to  review  the 
prequestions  again,  press  'HELP'  and  they  will  be  displayed  again.  A press 
of  the  'NEXT'  key  will  return  you  to  the  passage. 


Some  passages  are  contained  on  two  pages  (two  separate  computer  displays). 
You  will  always  begin  on  page  one.  If  the  passage  continues  on  the  next 
page,  this  will  be  indicated  at  the  bottom  of  the  first  page.  The  second 
page  of  a two-page  passage  will  be  displayed  when  you  press  the  'NEXT'  key. 

If  you  wish  to  return  to  the  first  page  from  the  second  you  may  do  so  by 
pressing  the  'BACK'  key. 

WhiJ v you  are  reading  through  a passage  you  may  review  its  prequestions 
(by  pressing  'HELP')  and  switch  between  its  pages  (by  pressing  'NEXT'  or 
'BACK')  as  many  times  as  you  wish. 
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After  you  have  completed  the  passage  and  feel  like  you  can  answer  the 
prequestions,  you  may  Indicate  this  by  pressing  the  ' f'  (for  finished)  key. 
You  cannot  go  back  to  review  the  previous  prequestions  or  passage  after 
you  have  pressed  the  ' f ' key,  so  do  not  press  it  before  you  feel  you  are 
ready  to  do  so . 

After  you  press  the  ' f'  key,  the  prequestions  for  the  next  passage  will 
be  displayed  and  you  will  begin  the  procedure  again.  When  you  have  completed 
the  last  passage  and  press  the  ' f * key,  you  will  be  asked  to  take  the  final 
test . Remember  that  this  test  will  contain  the  prequestions  you  have  seen 
with  each  passage  stated  in  multiple  choice  form.  The  final  test  will  be 
given  in  paper  and  pencil  format.  The  proctor  will  give  you  a copy  of  the 
test . 


If  you  have  any  questions  please  direct  them  to  the  proctor.  You  can 
take  as  much  time  as  you  need  to  read  through  the  passages.  When  you  are 
ready  to  begin  the  passage  readings  press  the  ’NEXT'  key. 


Key  Board  Directions 


To  get  from  PREQUESTIONS  to  PASSAGE  press  NEXT 
To  get  from  PASSAGE  to  PREQUESTIONS  press  HELP 
To  get  from  PAGE  1 of  a PASSAGE  to  PAGE  2 press  NEXT 
To  get  from  PAGE  2 of  a PASSAGE  to  PAGE  1 press  BACK 
To  get  from  PASSAGE  to  PREQUESTIONS  of  NEXT  PASSAGE  press 
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Student  Directions 


The  study  you  will  be  a part  of  concerns  methods  to  help  students  in 
learning  from  written  materials  presented  on  a computer  display.  The 
particular  method  we  are  interested  in  is  the  use  of  questions  related  to 
the  materials. 

You  will  be  asked  to  read  eight  passages  presented  on  a computer  terminal. 
Both  before  and  after  each  passage  you  will  be  given  four  questions.  These 
questions  request  information  which  is  either  contained  in  or  can  be  inferred 
from  the  passage.  (From  now  on  the  questions  preceding  a passage  will  be 
called  prequestions  and  those  following  a passage  will  be  called  postquestions.) 
The  prequestions  are  open-ended.  They  are  designed  to  direct  your  learning 
as  you  read  the  passage.  The  postquestions,  on  the  other  hand,  are  in 
multiple  choice  format.  They  are  designed  to  serve  as  check  points  on  how 
well  you  are  learning  the  materials.  If  you  feel  reasonably  confident  about 
your  answers  to  both  pre-  and  postquestions,  then  you  have  a sign  that  you 
have  studied  a passage  adequately. 

There  is  a connection  between  the  pre-  and  postquestions.  Two  of  the 
postquestions  are  merely  restatements  of  two  of  the  prequestions  in  multiple 
choice  format.  The  other  two  postquestions  are  not  the  same  as  the  prequestions. 

When  you  have  completed  reading  all  eight  passages  and  working  through 
the  pre-  and  postquestions,  you  will  be  asked  to  take  a final  test.  The 
test  will  contain  both  multiple  choice  postquestions  you  have  seen  as  well 
as  multiple  choice  restatements  of  the  passage  prequestions  you  did  not  receive 
as  postquestions.  It  will  measure  how  much  you  remember  from  the  information 
you  were  directed  to  learn  (from  both  the  pre-  and  postquestions).  It 
will  also  conclude  the  study. 


You  will  not  be  allowed  to  take  any  notes  while  you  are  reading  through 
the  materials. 

The  passages  are  about  400  words  each.  Each  concerns  different  and 
unrelated  topical  areas.  Topics  chosen  are  from  the  humanities,  social 
sciences,  physical  sciences,  and  biological  sciences. 

(As  you  read  through  the  rest  of  the  directions,  you  may  wish  to  refer 
to  the  Key  Board  Directions  summarized  on  the  next  page.) 

Passage  prequest  ions  will  be  displayed  on  the  terminal  before  you  see 
their  related  passage.  After  reading  and  familiarizing  yourself  with  these 
prequestions,  you  can  go  on  to  the  passage  by  pressing  the  'NEXT'  key  on 
the  computer  terminal  keyboard.  Thereafter,  anytime  you  wish  to  review  the 
prequestions  again,  press  'HELP'  and  they  will  be  displayed  again.  A press 
of  the  'NEXT'  key  will  return  you  to  the  passage. 


Some  passages  are  contained  on  two  pages  (two  separate  computer  displays). 
You  will  always  begin  on  page  one.  If  the  passage  continues  on  the  next 
page,  this  will  be  indicated  at  the  bottom  of  the  first  page.  The  second 
page  of  a two-page  passage  will  be  displayed  when  you  press  the  'NEXT'  key. 


If  you  wish  to  return  to  the  first  page  from  the  second  you  may  do  so  by 
pressing  the  'BACK'  key. 

While  you  are  reading  through  a passage  you  may  review  its  prequestions 
(by  pressing  'HELP')  and  switch  between  pages  (by  pressing  'NEXT'  or  'BACK') 
as  many  times  as  you  wish. 

After  you  have  completed  the  passage,  and  feel  like  you  can  answer  the 
prequestions,  you  mav  indicate  this  by  pressing  the  'f'  (for  finished)  key. 

You  cannot  go  back  to  review  the  prequestions  or  the  passage  after  you  have 
pressed  the  ' f 1 ke_y_,  so  do  not  press  it  before  you  feel  you  are  ready  to  do  so. 

After  you  press  the  ' f ' key  the  first  postquestion  will  be  displayed. 

Each  postquestion  will  be  displayed  one  at  a time.  Please  indicate  your 
answers  to  postquestions  by  pressing  the  letter  (A,B,C,D  or  E)  of  the  answer 
you  believe  is  correct.  The  answer  you  choose  will  be  written  at  the  bottom 
of  the  screen.  While  the  question  is  displayed,  if  you  change  your  mind 
about  an  answer,  just  press  the  'ERASE'  key  and  then  press  your  new  answer. 

When  you  have  completed  a postquestion  a press  of  the  'NEXT'  key  will 
display  the  next  one.  You  cannot  return  to  a previously  displayed  postquestion 
Also  you  cannot  go  on  to  the  next  postquestion  without  answering  the  one  you 
are  on.  When  you  have  completed  the  fourth  postquestion  a press  of  the  'NEXT' 
key  will  display  the  next  set  of  prequestions  and  you  will  begin  the  procedure 
again. 

When  you  press  the  'N  ' ' key  after  you  have  completed  the  last  post- 

question of  the  eighth  passage  you  will  be  asked  to  take  the  final  test. 
Remember  t hat  tills  test  will  contain  both  multiple  choice  postquestions  you 
have  seen  as  well_ as  multiple  choice  restatements  of  the  passage  prequestions 
you  did  not  receive  as  postquestions. 

The  final  test  will  be  given  in  paper  and  pencil  format.  The  proctor 
will  give  you  a copv  of  the  test. 

If  vou  have  any  questions  please  direct  them  to  the  proctor.  You  can 
take  as  much  t Ime  as  you  need  to  read  through  the  passages.  When  you  are 
ready  to  begin  the  passage  readings  press  the  'NEXT'  key  of  the  terminal 
keyboard. 

Key  Board  Directions 

To  get  from  PREQUESTIONS  to  PASSAGE  press  NEXT 

To  get  from  PASSAGE  to  PREQUESTIONS  press  HELP 

To  get  from  PAGE  1 of  a PASSAGE  to  PAGE  2 press  NEXT 

To  get  from  PAGE  2 of  a PASSAGE  to  PAGE  1 press  BACK 

To  get  from  PASSAGE  to  POSTQUESTIONS  press  f 
To  get  from  POSTQUF.STION  to  P0STQUF.ST10N  press  NEXT 

To  get  from  fourth  POSTQUESTION  to  PREQUESTIONS  OF  NEXT  PASSAGE  press  NEXT 

When  answering  POSTQUESTIONS,  to  change  an  answer  first  press  ERASE  and  then 
press  the  new  answer  (A,B,C,D  or  E) 


